{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CPSC 330 hw5\n",
    "\n",
    "**IMPORTANT NOTE:** This assignment, in its original form, is quite long. In Covid times, I am wary of overloading you. At the same time, I am also hesitant to delete useful material. My compromise is that I have marked some questions as optional and made them not part of your grade. If you have time, I recommend doing them! However, if you're short on time, you can skip them for now - later parts of the assignment don't depend on them. If you skip these questions, please look them over later as part of your exam studying - that content will still be fair game for exams. Solutions for these questions will be posted as part of the usual solutions posting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.dummy import DummyRegressor\n",
    "from sklearn.linear_model import Ridge, RidgeCV\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, cross_validate\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.compose import ColumnTransformer, TransformedTargetRegressor\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Non-sklearn\n",
    "from lightgbm import LGBMRegressor\n",
    "import shap\n",
    "\n",
    "# Add more imports below\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cross_validate_std(*args, **kwargs):\n",
    "    \"\"\"Like cross_validate, except also gives the standard deviation of the score\"\"\"\n",
    "    res = pd.DataFrame(cross_validate(*args, **kwargs))\n",
    "    res_mean = res.mean()\n",
    "\n",
    "    res_mean[\"std_test_score\"] = res[\"test_score\"].std()\n",
    "    if \"train_score\" in res:\n",
    "        res_mean[\"std_train_score\"] = res[\"train_score\"].std()\n",
    "    return res_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "rubric={points:5}\n",
    "\n",
    "Follow the [homework submission instructions](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing quality/quantity\n",
    "rubric={points:5}\n",
    "\n",
    "The TAs have reported a couple issues with the first few assignments: in some cases, submissions simply show the code output with no commentary; please write at least a sentence explaining your output in each question. In other cases, the TAs have come across multi-paragraph answers where a couple of sentences would have sufficed. Thus, we are now allocating the above points for well-structured answers of a reasonable length. In general, 1-3 sentences is good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment we'll be exploring a [dataset](https://www.kaggle.com/dgomonov/new-york-city-airbnb-open-data) of New York City Airbnb listings from 2019. As usual, you'll need to start by downloading the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('AB_NYC_2019.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_test = train_test_split(df, random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment we'll try to predict `reviews_per_month`, as a proxy for the popularity of the listing. Airbnb could use this sort of model to predict how popular future listings might be before they are posted, perhaps to help guide hosts create more appealing listings. In reality they might instead use something like vacancy rate or average rating as their target, but we do not have that available here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll take care of some missing values for you here. \n",
    "\n",
    "- It seems that listings with 0 reviews have `reviews_per_month` set to NaN instead of 0.\n",
    "- For listings without a listing name I set the name to the empty string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train.fillna({\"reviews_per_month\" : 0, \"name\" : \"\"})\n",
    "df_test  = df_test.fillna({\"reviews_per_month\" : 0, \"name\" : \"\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = 'reviews_per_month'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_train.drop(columns=[target_name])\n",
    "y_train = df_train[target_name]\n",
    "\n",
    "X_test  = df_test.drop(columns=[target_name])\n",
    "y_test  = df_test[target_name]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = y_train.fillna(value=0)\n",
    "y_test  = y_test.fillna(value=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = ['neighbourhood', 'neighbourhood_group', 'room_type']\n",
    "drop_features        = ['id', 'name', 'host_id', 'host_name', 'last_review', 'number_of_reviews']\n",
    "numeric_features     = ['latitude', 'longitude', 'price', 'minimum_nights', 'calculated_host_listings_count', 'availability_365']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure all columns are accounted for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert sorted(X_train.columns) == sorted(categorical_features + drop_features + numeric_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check for missing values (I already took care of some):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems they are only present in `host_name` and `last_review`, both of which we are dropping, so we can skip imputation here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer([\n",
    "    ('scale',  StandardScaler(), numeric_features),\n",
    "    ('ohe', OneHotEncoder(handle_unknown='ignore'), categorical_features)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor.fit(X_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe_feature_names = list(preprocessor.named_transformers_['ohe'].get_feature_names(categorical_features))\n",
    "new_columns = numeric_features + ohe_feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_enc = pd.DataFrame(preprocessor.transform(X_train).toarray(), index=X_train.index, columns=new_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_enc.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll try training some models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dr = DummyRegressor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(cross_validate(dr, X_train, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, the score here is the $R^2$. Let's try linear regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = make_pipeline(preprocessor, Ridge())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(cross_validate(lr, X_train, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And a LightGBM regressor (using this because it runs much faster than `RandomForestRegressor`!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lgbm = make_pipeline(preprocessor, LGBMRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_validate_std(lgbm, X_train, y_train, return_train_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1: data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(a)\n",
    "rubric={points:5}\n",
    "\n",
    "Above I dropped the feature `number_of_reviews`. And yet, knowing the number of reviews seems quite useful in predicting the number of reviews per month. Why would it be problematic to leave this feature in?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 1(b)\n",
    "\n",
    "I noticed there were no missing values in the columns we're using, so I didn't put any imputation into the pipeline. In what scenario could that result in a problem down the line?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(c)\n",
    "rubric={points:5}\n",
    "\n",
    "After one-hot encoding, the number of columns increased significantly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(numeric_features) + len(categorical_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(new_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the categorical variables printed below, how many new columns were generated?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 1(d)\n",
    "\n",
    "In the previous part you (hopefully) found that the vast majority of categories come from the \"neighborhood\" column. Let's look at how many of each category we have in the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['neighbourhood'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perhaps it doesn't make sense to have separate columns for some of these very rare cases. The code below modifies the preprocessor so that the categories are specified inside the `OneHotEncoder`. It uses the 30 most frequent neighbourhoods in the training set as the fixed categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = df_train['neighbourhood'].value_counts()[:30].index.values\n",
    "categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor_cats = ColumnTransformer([\n",
    "    ('scale',  StandardScaler(), numeric_features),\n",
    "    ('ohe_cats', OneHotEncoder(handle_unknown='ignore', categories=[categories]), [\"neighbourhood\"]),\n",
    "    ('ohe', OneHotEncoder(handle_unknown='ignore'), [\"neighbourhood_group\", \"room_type\"])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then train two linear regression models with small `alpha` values, one with the original preprocessor and one with the modified preprocessor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_small = make_pipeline(preprocessor, Ridge(0.01))\n",
    "cross_validate_std(lr_small, X_train, y_train, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_small_cats = make_pipeline(preprocessor_cats, Ridge(0.01))\n",
    "cross_validate_std(lr_small_cats, X_train, y_train, return_train_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In class I briefly mentioned that for our linear models (`LogisticRegression` and `Ridge`), having more features can be considered as having a more complex model because we have more coefficients. Given this, discuss the results above with respect to the fundamental tradeoff. Do the results align with your expectations?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier we dropped the `name` column, but perhaps the names contain useful information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train[\"name\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, maybe listings with the word \"new\" are more popular than listings with the word \"rustic\". Let's update the feature preprocessing so that `name` is not dropped, but instead encoded with `CountVectorizer`. Let's update the preprocessor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor_text = ColumnTransformer([\n",
    "    ('scale',  StandardScaler(), numeric_features),\n",
    "    ('ohe', OneHotEncoder(handle_unknown='ignore'), categorical_features),\n",
    "    ('text', CountVectorizer(max_features=100, stop_words='english'), \"name\")\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at the transformed training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor_text.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_feature_names = ['name_' + word for word in preprocessor_text.named_transformers_['text'].get_feature_names()]\n",
    "new_columns_text = new_columns + word_feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_enc_text = pd.DataFrame(data=preprocessor_text.transform(X_train).toarray(), columns=new_columns_text, index=X_train.index)\n",
    "X_train_enc_text.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a lot of features generated by the OHE and the word counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe_feature_names[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_feature_names[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, it seems we now have a feature called `neighbourhood_Williamsburg` and a feature called `name_williamsburg`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_enc_text[['neighbourhood_Williamsburg', 'name_williamsburg']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(e)\n",
    "rubric={points:5}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What is the difference between our features  `neighbourhood_Williamsburg` and `name_williamsburg`? \n",
    "- Is it possible that one could be zero and the other nonzero?\n",
    "- Is it redundant to have both?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 1(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we explore the target values. The number of reviews per month ranges quite a bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.hist(bins=100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We decide to log-transform these targets, because decide we care more about relative than absolute error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_log = np.log(y_train)\n",
    "y_test_log  = np.log(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "y_train_log.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no, an error!\n",
    "\n",
    "- Explain the issue.\n",
    "- Fix the issue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: model selection\n",
    "\n",
    "#### 2(a)\n",
    "rubric={points:15}\n",
    "\n",
    "- Use `GridSearchCV` or `RandomizedSearchCV` to tune a `LGBMRegressor` pipeline on this problem, using `preprocessor_text` with un-transformed y-values. \n",
    "- Tune `num_leaves` of the `LGBMRegressor` and `max_features` of the `CountVectorizer` at least (you're welcome to include more hyperparameters if you want).\n",
    "- What hyperparameters were chosen and what is the final score? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(b)\n",
    "rubric={points:5}\n",
    "\n",
    "Briefly comment on the \"error bars\" from the different folds of cross-validation. Do you trust the ordering of the hyperparameters from the previous part? Briefly discuss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(c)\n",
    "rubric={points:5}\n",
    "\n",
    "Run your chosen pipeline on the test set. How does your test error compare to your cv error? Do you think you overfit the (cross-)validation set? Briefly discuss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: feature importances\n",
    "\n",
    "For this exercise we'll consider the random forest created below, which performs somewhat decently (though still not great):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestRegressor(max_depth=20, max_features=20, n_estimators=20, random_state=20)\n",
    "rf_ttr = TransformedTargetRegressor(rf, func=np.log1p, inverse_func=np.expm1)\n",
    "rf_ttr_pipe = make_pipeline(preprocessor_text, rf_ttr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_validate_std(rf_ttr_pipe, X_train, y_train, return_train_score=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(a)\n",
    "rubric={points:10}\n",
    "\n",
    "Look at the feature importances for this random forest pipeline trained on the training set. What features seem to be driving your predictions most?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(b)\n",
    "rubric={points:5}\n",
    "\n",
    "For the two most important features in the above model, do you think increasing these features increases or decreases the predicted number of reviews per month? Briefly justify your answer. **Maximum 2 sentences.**\n",
    "\n",
    "Note: the results from part (a) don't tell you the answer here - I'm asking you to use your domain knowledge about short-term vacation rentals. What do you think makes a rental attractive?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(c)\n",
    "rubric={points:10}\n",
    "\n",
    "Use SHAP to try to answer the above question more definitively, for predictions on the training set. You will likely need to take a subset of the training examples for speed when computing the SHAP values. **Maximum 1 paragraph.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 3(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code adds a column of random noise to `X` and re-trains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_noise = np.random.randn(X_train_enc_text.shape[0],1)\n",
    "random_noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_words_noise = pd.concat((X_train_enc_text, pd.DataFrame(random_noise, columns=['noise'], index=X_train_enc_text.index)), axis=1)\n",
    "X_train_words_noise.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_noise = RandomForestRegressor(max_depth=20, max_features=20, n_estimators=20, random_state=20)\n",
    "rf_noise.fit(X_train_words_noise, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can look at the feature importances of this noise column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(data=rf_noise.feature_importances_, index=X_train_words_noise.columns, columns=[\"Importance\"]).tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But wait, we know this feature can't possibly be useful - why is the importance of the random noise feature non-zero (and in fact larger than for some real features)?  **Maximum 2 sentences.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(e)\n",
    "rubric={points:10}\n",
    "\n",
    "One of the numeric features is `price`, which is presumably in U.S. dollars per night. We can fit a `Ridge` model to the data and look at the coefficient corresponding to `price`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = make_pipeline(preprocessor_text, Ridge(alpha=100))\n",
    "lr.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coefs = pd.DataFrame(data=lr[1].coef_, index=new_columns_text, columns=[\"Coefficient\"])\n",
    "coefs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "price_coef = coefs.loc[\"price\"].values[0]\n",
    "price_coef"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, we can interpret a linear regression coefficient as the increase/decrease in the predicted target per unit change of the feature: for example, increasing the price by \\\\$1/night would decrease the predicted reviews/month by the above amount (decrease because of the negative coefficient).\n",
    "\n",
    "However, that would be incorrect because we scaled the numeric features in the preprocessing pipeline, as discussed in lecture. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = preprocessor_text.named_transformers_['scale']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking into account the scaling, find the relationship implied by the model. Your answer should be in the units of (reviews per month) / (dollars per night); that is, your answer should be in the form of \"Increasing the price by \\$1 increases/decreases the predicted target by ___ reviews/month\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 3(f)\n",
    "\n",
    "Perform an open-ended exploration of the feature importances using the model of your choosing. What is the most fun or interesting result you found? You could examine, for example:\n",
    "\n",
    "- Different room types\n",
    "- The importance of various numeric features\n",
    "- The \"most positive\" and \"most negative\" words.\n",
    "\n",
    "Recommended time spent: 5-15 min."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(g)\n",
    "rubric={points:10}\n",
    "\n",
    "Given your exploration in the previous part, can you draw conclusions about the world, such as the desireability of various neighbourhoods in New York? Could you now give advice to a friend on what words are best to use in an Airbnb listing? My answer, as you may have guessed, is no! (Or, at least, not without extreme caution and/or statistical training that goes beyond the scope of this course.) Give **3 reasons** why you might be wrong if you drew such conclusions about the world from the above analysis. **Maximum 1 sentence** per reason."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: very short answer questions\n",
    "rubric={points:25}\n",
    "\n",
    "Answer each of the following questions in **at most 1 sentence**. Each one is worth 5 points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. If `neighbourhood_group` is aready a feature, why might `neighbourhood` still be worth including?\n",
    "2. If `neighbourhood` is aready a feature, why might `neighbourhood_group` still be worth including?\n",
    "3. If you had an enormous training set and could only keep one of `neighbourhood` or `neighbourhood_group`, which one would you keep, and why?\n",
    "4. How does the size of your training set influence how much you \"trust\" your feature importances?\n",
    "5. If you wanted to find the \"most positive\" and \"most negative\" words in terms of reviews per month, would you use `Ridge` or `RandomForestRegressor`? Assume you're using only scikit-learn (no SHAP, etc.).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission to Canvas\n",
    "\n",
    "**IF YOU ARE WORKING WITH A PARTNER** please form the group before submitting - see instructions [here](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md#partners).\n",
    "\n",
    "When you are ready to submit your assignment do the following:\n",
    "\n",
    "1. Run all cells in your notebook to make sure there are no errors by doing `Kernel -> Restart Kernel and Clear All Outputs` and then `Run -> Run All Cells`.\n",
    "2. Save your notebook.\n",
    "3. Convert your notebook to `.html` format using the `convert_notebook()` function below **or** by `File -> Export Notebook As... -> Export Notebook to HTML`.\n",
    "4. Run the code `submit()` below to go through an interactive submission process to Canvas.\n",
    ">For this step, you will need a Canvas *Access Token* token. If you haven't already got one, log-in to Canvas, click `Account` (top-left of the screen), then `Settings`, then scroll down until you see the `+ New Access Token` button. Click that button, give your token any name you like and set the expiry date to Dec 31, 2020. Then click `Generate token`. Save this token in a safe place on your computer as you'll need it for all assignments. Treat the token with as much care as you would an important password. \n",
    "\n",
    "Note: for those having trouble with the Jupyter widgets and the dropdowns: if you add the argument `no_widgets=True` to your `submit` call, it should let you do a text-based entry of your key and avoid the dropdowns altogether. If this doesn't work, you probably need to upgrade to the latest version of `canvasutils` with `pip install canvasutils -U` from your terminal with your environment activated.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from canvasutils.submit import submit, convert_notebook\n",
    "\n",
    "# Note: the canvasutils package should have been installed as part of your environment setup - \n",
    "# see https://github.com/UBC-CS/cpsc330/blob/master/docs/setup.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert_notebook(\"hw5.ipynb\", \"html\")  # uncomment and run when you want to try convert your notebook to HTML (or you can convert manually from the File menu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit(course_code=53561, token=False)  # uncomment and run when ready to submit "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "name": "_merged",
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "438px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
